Fusion-based wet road surface detection

ABSTRACT

A method for determining wetness on a path of travel. A surface of the path of travel is captured by at least one image capture device. A plurality of wet surface detection techniques is applied to the at least one image. An analysis for each wet surface detection technique is determined in real-time of whether the surface of the path of travel is wet. Each analysis independently determines whether the path of travel is wet. Each analysis by each wet surface detection technique is input to a fusion and decision making module. Each analysis determined by each wet surface detection technique is weighted within the fusion and decision making module by comprehensive analysis of weather information, geology information, and vehicle motions. A wet surface detection signal is provided to a control device. The control device applies the wet surface detection signal to mitigate the wet surface condition.

BACKGROUND OF INVENTION

An embodiment relates generally to detection of water and water filmthickness on a path of travel.

Precipitation on a driving surface causes several different issues for avehicle. For example, water on a road reduces the coefficient offriction between the tires of the vehicle and the surface of the roadresulting in vehicle stability issues. Detection of precipitation on aroad of travel is typically determined by a host vehicle sensing forprecipitation on the road utilizing some sensing operation which occurswhen the precipitation is already impacting the vehicle operation suchas detecting wheel slip. As a result, the vehicle must monitor its ownoperating conditions (e.g., wheel slip) against dry pavement operatingconditions for determining whether precipitation is present. As aresult, such systems may wait for such a condition to occur or mayintroduce excitations to the vehicle for determining whether thecondition is present (e.g., generating sudden acceleration to the drivenwheels for invoking wheel slip if the precipitation is present).

Moreover, individual sensing techniques typically focus on a singleconcept for detecting wetness of a road where each respective techniqueworks well under certain environmental conditions (e.g., a certain levelof water), but may be deficient outside of those specific environmentalconditions. Therefore, it would be beneficial to have a technique thatcan adapt to different environmental conditions and provide reliableresults.

SUMMARY OF INVENTION

An advantage of an embodiment is the detection of a wet surface of apath of travel for either alerting a driver of the wet surface conditionand/or actuating a vehicle control system for mitigating the effects ofthe wet surface using a plurality of individual techniques where theresults of the individual techniques are fused and weighted forenhancing the assessment and reliability of detecting the wet surfacedetection. The technique described herein cooperatively utilizes aplurality of vision-based wet surface detection techniques where each ofthe individual results are input to a fusion and detection module. Eachof the results are weighted and the results are cooperatively fused togenerate an output indicating whether the path of travel is wet or notwet. A condition assessment module determines weighting factors as afunction of estimated water depth level and vehicle speed. The estimatedwater depth level is determined as a function of rain condition data andpath of travel topology data. The weighting and fusion of the resultprovides an enhanced confidence level of detecting water on the path oftravel than just utilizing individual techniques.

An embodiment contemplates a method for determining wetness on a path oftravel. At least one image of a surface of the path of travel iscaptured by at least one image capture device. The at least one imagecapture device focusing at the surface where water is expected as avehicle travels along the path of travel. A plurality of wet surfacedetection techniques is applied, by a processor, to the at least oneimage. A determination is made in real-time an analysis for each wetsurface detection technique of whether the surface of the path of travelis wet. Each analysis independently determines whether the path oftravel is wet. Each analysis by each wet surface detection technique isinput to a fusion and decision making module. Each analysis determinedby each wet surface detection technique is weighted within the fusionand decision making module. A wet surface detection signal is providedto a control device. The control device applies the wet surfacedetection signal to mitigate the wet surface condition.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 an exemplary perspective view of a vehicle scene on a wet surfacecaptured by a camera.

FIG. 2 illustrates a block diagram of a wet surface detection system.

FIG. 3 is an exemplary perspective view of a vehicle surround havingsurround view coverage.

FIG. 4 illustrates an exemplary image captured of a mirrored wetsurface.

FIG. 5 illustrates an exemplary an image captured of rearward splashoccurs on wet surface.

FIG. 6 illustrates an exemplary image captured of side tire splash onwet surface.

FIG. 7 illustrates an image when tire tracks are generated on a wetsurface.

FIG. 8 illustrates a block diagram of the proposed wet surface detectiontechnique based on fusion and weighting scheme

FIG. 9 illustrates a flow diagram for determining weighting factorswithin the condition assessment model.

DETAILED DESCRIPTION

There is shown in FIG. 1, a vehicle 10 traveling along a vehicle road12. Precipitation 19, in the form of water, is shown disposed on thevehicle road 12. The precipitation 19 is often displaced by the vehiclewheel 14 and tires 16 mounted on a rim 18 of the wheel 14 as the tiresrotate over the wet surface on the vehicle road 12 or other path oftravel. Therefore, it is advantageous to know when the vehicle will betraveling along a wet vehicle road 12 so that issues resulting fromwater, such as loss of traction or engine degradation resulting fromwater entering exterior air intake vents can be identified and negated,Identifying the water on the vehicle road 12 can assist the vehicle indetermining an appropriate countermeasure for negating loss of tractionand other negative effects that water can have on the vehicle. It shouldbe understood that although an automobile is used herein for exemplarypurposes, the embodiments described herein can be applied to other typesof systems aside from automobiles where detection of a wet surfacecondition is desired. Examples of vehicles that are other thanautomobiles that can utilize this system include, but are not limitedto, rail systems, planes, off-road sport vehicles, robotic vehicles,motorcycles, bicycles, farm equipment, and construction equipment.

Precipitation 19 on the vehicle road 12 can result in a reduction oftraction when driving on the wet surface. The precipitation 19 disposedon the vehicle road 12 lowers the coefficient of friction between thevehicle tires and the vehicle road 12. As a result, traction between thevehicle tires and the vehicle road 12 is lowered. Detecting water on theroad can assist the vehicle in determining the appropriate mitigationtechnique for minimizing any loss of traction by various mitigationtechniques that include, but are not limited to, warning the driver tolower the vehicle speed to one that is conducive to the environmentalconditions; actuating automatic application of the vehicle brake using avery low braking force to minimize the precipitation formed on thebraking surfaces of the braking components; deactivation or restrictingthe activation of some advanced driver assistance features such asadaptive cruise control, lane centering, and collision avoidance whileprecipitation is detected; or notification to the driver to maintain agreater stopping distance to a lead vehicle.

FIG. 2 illustrates a block diagram of various hardware devices andsystems used by the respective techniques to detect wet surfaces and tocounteract wet surface conditions. A plurality of vehicle-based imagecapture devices 20 are mounted on the vehicle for capturing imagesaround the vehicle. The plurality of vehicle based image capture devices20 may be mounted on the front, rear, and sides of the vehicle. FIG. 3illustrates an exemplary 360 degree surround view coverage for detectingobjects around the vehicle. Each of the image-based capture devices arecooperatively used to detect and identify objects on each side of thevehicle. The image-based capture devices 20 include, but are not limitedto, a front view camera 22 mounted to the front of the vehicle capturingimage forward and partially to the sides of the vehicle, a driver's sidecamera 24 capturing images on the driver side of the vehicle, apassenger's side camera 26 capturing images on the passenger side of thevehicle, and a rearward facing camera 28 capturing images rearward andto the side of the vehicle.

Referring again to FIG. 2, a processor 30 processes the images capturedby the image capture devices 20. The processor 30 analyzes images anddata to determine whether water is present on the surface of the path oftravel based on various water detection techniques. Such techniques mayinclude a mirrored light image analysis technique, a tire rearwardsplash analysis technique, a tire side splash analysis technique, and atire track analysis technique. Each of the respective techniques aredescribed in co-pending application (Ser. No. 14/568,656) dated Dec. 12,2014 entitled “Systems And Method For Determining A Condition Of A RoadSurface”; co-pending application (Ser. No. 14/957,953) dated Dec. 3,2015 entitled “Vision-Based Wet Road Surface Condition Detection UsingTire Rearward Splash”; co-pending application (Ser. No. 14/957,998)dated Dec. 3, 2015 entitled “Vision-Based Wet Road Surface ConditionDetection Using Tire Side Splash”; and co-pending application (Ser. No.14/957,983) dated Dec. 3, 2015 entitled “Vision-Based Wet Road SurfaceCondition Detection Using Tire Tracks”, which are each incorporated byreference in their entirety. The results of each technique are thenfused utilizing a weighting scheme to cooperatively determine whetherthe path of travel is a wet surface. The fusing and weight schemeprovides enhanced reliability and robustness for determining whether thepath of travel is wet or not.

The processor 30 may be part of an existing system, such as tractioncontrol system or other system, or can be a standalone processordedicated to a road condition detection function, which may have inputsfrom different sources such as image capture devices 22, CAN bus signals(eg. vehicle speed, temperature, humidity, etc.), WiFi weather, and pathof travel topology information from other modules or devices.

The processor 30 may be coupled to one or more output devices such as acontroller 32 for initiating or actuating a control action based on theanalysis applied by the processor. One or more countermeasures may beactuated for mitigating the effect that the water may have on theoperation of the vehicle.

The controller 32 may be part of the vehicle subsystem or may be used toenable a vehicle subsystem for countering the effects of the water. Forexample, in response to a determination that the road is wet, thecontroller 32 may enable an electrical or electro-hydraulic brakingsystem 34 or similar where a braking strategy is readied in the eventthat traction loss occurs. In addition to preparing a braking strategy,the braking system may autonomously apply a light braking force, withoutawareness to the driver, to remove water from the vehicle brakes oncethe vehicle enters the water. Removal of water build-up from the wheelsand brakes maintains an expected coefficient of friction between thevehicle brake actuators and the braking surface of the wheels whenbraking by the driver is manually applied.

The controller 32 may control a traction control system 36 whichdistributes power individually to each respective wheel for reducingwheel slip by a respective wheel when a respective amount of water isdetected on the surface of the path of travel such as in the case ofhydroplaning.

The controller 32 may control an advanced driver assistance system (forexample, cruise control system, adaptive cruise control system, lanefollowing system, lane change system, evasive/assist steering maneuversystem, automated emergency braking system, etc.) which can deactivatethe system functionality or restrict the activation of the system whenwater is detected on the surface of the path of travel.

The controller 32 itself may be an advanced driver assistance systemwhich is designed to automatically adjust its system functionality toaccommodate the surface wetness by integrating wet surface signal intoits controller design process and perform safely when water is detectedon the surface of the path of travel.

The controller 32 may control a driver information system 40 forproviding warnings to the driver of the vehicle concerning water that isdetected on the vehicle road. Such a warning actuated by the controller32 may alert the driver to the approaching water on the surface of thepath of travel and may recommend that the driver lower the vehicle speedto a speed that is conducive to the current environmental conditions, orthe controller 32 may actuate a warning to maintain a safe drivingdistance to the vehicle forward of the driven vehicle. It should beunderstood that the controller 32, as described herein, may include oneor more controllers that control an individual function or may control acombination of functions.

The controller 32 may further control the actuation of automaticallyopening and closing air baffles 42 for preventing water ingestion intoan engine of the vehicle. Under such conditions, the controller 32automatically actuates the closing of the air baffles 42 when water isdetected to be present on the surface of the path of travel in front ofthe vehicle and may re-open the air baffles when water is determined tono longer be present on the surface.

The controller 32 may further control the actuation of a wirelesscommunication device 44 for autonomously communicating the wet pavementcondition to other vehicles utilizing a vehicle-to-vehicle orvehicle-to-infrastructure communication system.

The controller may further provide the wet surface signal alerts to adriver of the vehicle against a use of advanced driver assistancesystems.

The various techniques described above each provide a novel approach asto determining wet surface. Each of the following figures representsexemplary images where the respective techniques are capable ofdetecting water on the surface based on the disbursement of water orreflection of light in the image. For example, FIG. 4 illustrates animage captured of a mirrored surface where ice or a wet road is detectedby the mirrored light image analysis technique. FIG. 5 illustrates animage captured when a rearward splash occurs that is detected by therearward splash analysis technique. FIG. 6 illustrates an image capturedwhen a side tire splash occurs that is detected by the side splashanalysis technique. FIG. 7 illustrates an image when tire tracks aregenerated on a wet surface that is detected by the tire track analysistechnique. Each of the techniques described earlier is proficient atidentifying water on the path of travel when water displacement ornon-displacement is present under certain conditions. Therefore, thefollowing technique describes a process to use each of the techniquescooperatively for enhancing reliability in detecting a wet vs non-wetsurface.

FIG. 8 illustrates a block diagram of the wet road condition based onweighting scheme. Analysis of a road surface is applied by each of therespective techniques described earlier such as mirrored light imageanalysis T₁, tire rearward splash analysis T₂, tire side splash analysisT₃, tire track analysis T₄. Each of the respective techniques outputs awet signal (e.g., 1) or a non-wet signal (e.g., 0). Each output signaldetermined by each technique is input to a fusion in decision-makingmodule 70.

The fusion and decision-making module 70 utilize each of the respectivetechniques for generating a final decision related to surface conditionof the path of travel. However, each of the respective inputs to thefusion decision-making module 70 may not be equally weighted. That is, atechnique is applied for allocating a weight to each respective inputthat is dynamically determined based on assessments from a conditionassessment module 72.

The condition assessment module 72 utilizes a plurality of environmentalconditions, geology conditions, and vehicle operating conditions fordetermining weighting factors that are applied to each of the respectiveinputs within the fusion decision-making module 70.

The environmental condition includes rain condition information 74. Suchdata may include, but is not limited to, rain status (e.g., rain orno-rain), strength of rain (e.g., light, medium, heavy), rain duration,elapsed time since rain stopped, amount of rain. Such information may beobtained from various sources that include, but are not limited to, RealTime WiFi, weather Apps, cloud information.

The geology conditions include road topology 76. Such data may include,but is not limited to, road type (e.g., flat/slope/low-lying area), roadelevation, and road grade. Such information may be obtained from varioussources that include, but are not limited to, a 3-dimensional maps, GPS,and road grade estimation algorithms.

The vehicle operating conditions may include, but is not limited to,vehicle longitudinal speed 78. Such information may be obtained fromvarious sources that include, but are not limited to, vehicle messagescommunicated through the vehicle CAN (obtained from speed sensors, wheelsensors, vehicle engine sensors, and other powertrain components), andGPS devices.

FIG. 9 illustrates a flow diagram for determining weights of eachindividual detection technique within the condition assessment model 70.Rain condition information 74 and path of travel topology information 76are two important factors that influence the water depth on a path oftravel. So rain condition information and path of travel topologyinformation are evaluated and cooperatively used to estimate the waterdepth level on the surface of the path of travel. The water depth may becategorized into various levels such as minimum, shallow, medium, anddeep. For example, a water thickness<0.5 mm is categorized as a minimumlevel, a water thickness between 0.5 mm-1.0 mm is categorized as ashallow level; a water thickness between 1.0 mm-2.5 is categorized as amedium level; and water thickness>2.5 mm is categorized as a deep level.It should be understood that the various classification levels and theirassociated water depth levels are exemplary and various classificationand water depth level ranges may be utilized. The water depth level onthe path of travel can be estimated using a machine learning classifierincluding, but not limited to, a Neural network classifier or a Bayesiannetwork classifier. The network weights can be trained in advance usinginput training data of rain information and road topology information.Alternatively, logic deduction based on empirical data may be used toestimate the water depth level on the surface of the path of travel.After the estimated water depth level 80 on the path of travel isestimated, the estimated water depth level and the speed of the vehicleare then cooperatively utilized to determine a weight factor for eachrespective technique within a weight calculation module 81. A weightfactor of each respective technique indicates a confidence level for thedetection performance of that particular technique under currentenvironmental conditions. For example estimated water depth levels (D)may be represented by the following expression:

D={d1=minimal,d2=shallow,d3=medium,d4=deep}.

The vehicle speed level (V) may be represented by the followingexpression:

V={v1=low,v2=medium,v3=high}.

The respective techniques mirrored light image technique, tire rearwardsplash technique, tire side splash technique, and tire track techniqueeach may be represented by {T1, T2, T3, T4} respectively. As a result,weight factors may be determined for each respective technique asfollows:

W={w1(for T1),w2(for T2),w3(for T3),w4(for T4)}.

Therefore, the weighting factors can be calculated as a conditionalprobability for each respective technique T_(i) when the estimated waterdepth level D and vehicle speed V are known. The representation is asfollows:

W=P(T _(i) |D,V)

where P is a conditional probability, T_(i) is a respective technique(e.g., {T1, T2, T3, T4}, D is the water depth level that can take on arespective value {shallow, medium, deep}, and V is the velocity of thevehicle that can take on a respective value {low, medium, high}. As aresult, a determination is made whether the path of travel is either wet(e.g., 1) or not wet (e.g., 0). If the determination is made that thepath of travel is wet, then the respective signal is provided to awarning/control application module 84 to either alert the driver of thewetness on the path of travel or the vehicle may utilize thisinformation to enable or disable vehicle operations or vehicle stabilityoperations.

The following table illustrates exemplary weighting factors assigned foreach technique based on empirical data. A confidence level is determinedfor each technique.

D V T1 T2 T3 T4 shallow low 90%  9% 0.5%   0.5%   shallow medium 70% 29%0.5%   0.5%   shallow high 50% 48% 1% 1% medium low 48% 48% 1% 1% mediummedium 30% 60% 5% 5% medium high  5% 85% 5% 5% deep low  1% 30% 40% 50%  deep medium 0.1%  29.5%   35%  35%  deep high 0.1%  19.5%   40% 40% As illustrated in this exemplary table, when the water is shallow andspeed is low, technique T1 works best as indicated by the confidencelevel. As the water depth increases and speed increases, the techniqueT2 works best. As the water depth is determined to be deep, thentechniques T3 and T4 work best. Based on this empirical sensing data, aprobability can be obtained using the weighting factor determined foreach technique derived by P(T_(i)|D,V).

In response to determining a weight for each technique, voting by aweighted voting module 82 can be determined as follows:

Voting number=w1*T1+w2*T2+w3*T3+w4*T4

where T_(i) (i=1, 2, 3, 4) takes on either 1 (wet) or 0 (non-wet). Rulesmay be made such that if the voting number is larger than apredetermined threshold (e.g., 50%), then the determination is made thatthe surface is wet; otherwise, the surface is non-wet.

The warning/control application module 84 may be an output device wherethe warning to the driver may be a visual, audible, or haptic output tonotify the user of the wet surface condition on the path of travel.Moreover, the warning/control application module may be control devicethat utilizes the information to actuate vehicle controls to counteractthe wetness on the path of travel.

While certain embodiments of the present invention have been describedin detail, those familiar with the art to which this invention relates,will recognize various alternative designs, filtering processes andembodiments for practicing the invention as defined by the followingclaims.

What is claimed is:
 1. A method for determining wetness on a path oftravel, the method comprising the steps of: capturing at least one imageof a surface of the path of travel by at least one image capture device,the at least one image capture device focusing at the surface wherewater is expected as a vehicle travels along the path of travel;applying a plurality of wet surface detection techniques, by aprocessor, to the at least one image; determining in real-time ananalysis for each wet surface detection technique of whether the surfaceof the path of travel is wet, each analysis independently determiningwhether the path of travel is wet; inputting each analysis by each wetsurface detection technique to a fusion and decision making module;weighting each analysis determined by each wet surface detectiontechnique within the fusion and decision making module; and providing awet surface detection signal to a control device, the control deviceapplying the wet surface detection signal to mitigate the wet surfacecondition.
 2. The method of claim 1 wherein the plurality of wet surfacedetection techniques includes at least a rearward tire splash analysistechnique.
 3. The method of claim 1 wherein the plurality of wet surfacedetection techniques includes at least a side tire splash analysistechnique.
 4. The method of claim 1 wherein the plurality of wet surfacedetection techniques includes at least a tire track analysis technique.5. The method of claim 1 wherein the plurality of wet surface detectiontechniques includes at least a mirrored light image analysis technique.6. The method of claim 1 wherein each analysis determined by each wetsurface detection technique is normalized representing one of a wetsurface or a non-wet surface.
 7. The method of claim 1 wherein weightingis determined by a condition assessment module, wherein the conditionassessment module provides the weights that are to be applied to eachtechnique to the fusion and decision making module.
 8. The method ofclaim 7 wherein the weighting as determined by the condition assessmentmodule is a function of the rain condition data, path of travel topologydata, and vehicle speed data.
 9. The method of claim 8 wherein thecondition assessment module determines a water depth level on the pathof travel as a function of the rain condition data and the road topologydata.
 10. The method of claim 9 wherein the condition assessment moduledetermines the weighting for each respective wet surface detectiontechnique as a function of the water depth level on the path of traveland the vehicle speed.
 11. The method of claim 7 wherein the conditionassessment module utilizes a Neural Network to determine the water depthlevel on the path of travel and the weighting factors.
 12. The method ofclaim 7 wherein the condition assessment module utilizes a BayesianNetwork to determine the water depth level on the path of travel and theweighting factors.
 13. The method of claim 7 wherein the conditionassessment module utilizes logic deduction to determine the water depthlevel on the path of travel.
 14. The method of claim 7 wherein theweighting is determined by the following equation:W=P(T _(i) |D,V) where P is a conditional probability, T_(i) is arespective wet road surface detection technique, D is the water depthlevel, and V is the velocity of the vehicle.
 15. The method of claim 14wherein the fusion and decision making module applies a weighted vote toeach analysis, the weighted vote determined as follows:Voting number=w1*T1+w2*T2+w3*T3+w4*T4 where T_(i) (i=1, 2, 3, 4) takeson either 1 (wet) or 0 (non-wet).
 16. The method of claim 1 wherein thecontrol device uses the wet surface detection signal to determine avehicle braking strategy to autonomous actuate a braking system.
 17. Themethod of claim 1 wherein the control device uses the wet surfacedetection signal to determine a traction control strategy toautonomously actuate a traction control system.
 18. The method of claim1 wherein the wet surface detection signal is provided to a wirelesscommunication system to alert other vehicles of the wetness on the pathof travel.
 19. The method of claim 1 wherein the control device uses thewet surface detection signal to alert a driver of a potential reducedtraction between vehicle tires and the surface as a result of theidentified wetness on the path of travel.
 20. The method of claim 1wherein control device uses the wet surface detection signal to alert adriver of the vehicle against a use of a driver assistance system. 21.The method of claim 1 wherein the control device uses the wet surfacedetection signal to autonomously modify a control setting of anautomated control feature in response to the identified wetness on thepath of travel.
 22. The method of claim 1 wherein the control deviceuses the wet surface detection signal to alert a driver to reduce avehicle speed in response to the identified wetness on the path oftravel.
 23. The method of claim 1 wherein the control device uses thewet surface detection signal to shut baffles on an air intake scoop of avehicle for preventing water ingestion in response to the identifiedwetness on the path of travel.